1. Field of the Invention
The present invention relates generally to electronic communication. More specifically, the present invention relates to the sharing of single-root input/output virtualization devices by multiple roots.
2. Description of the Related Art
Virtualization involves a way to run multiple environments on a single device or system. Using virtualization, extra processing power and/or storage on a device can be more efficiently used by sharing it between more than one environment. Each environment is known as a virtual machine (VM), and typically these systems are constructed in a manner that allows programs running within a virtual machine to operate without knowledge that the environment is sharing resources with other environments.
In addition to interoperability, virtual machines also must take into account security concerns. Typically, I/O virtualization solutions provide the same isolation that was found when the environment was operating on a separate physical machine. Isolation involves separation of memory space, input/output (I/O) streams, interrupts, and the ability to isolate control operations, I/O operations, and errors.
Computer architectures have advanced greatly over the years. Lately it is becoming more and more commonplace for chip designers to include external data interfaces, such as Universal Serial Bus (USB) interfaces into their motherboards. These interfaces are known as host controllers. The processor is typically then connected to the other components of the computer system via an input/output (I/O) interconnect system.
There are many different computer I/O interconnect standards available. One of the most popular over the years has been the peripheral component interconnect (PCI) standard. PCI allows the bus to act like a bridge, which isolates a local processor bus from the peripherals, allowing a Central Processing Unit (CPU) of the computer to run must faster.
Recently, a successor to PCI has been popularized, termed PCI Express (or, simply, PCIe). PCIe provides higher performance, increased flexibility and scalability for next-generation systems, while maintaining software compatibility with existing PCI applications. Compared to legacy PCI, the PCI Express protocol is considerably more complex, with three layers—the transaction, data link and physical layers.
In a PCI Express system, a root complex device connects the processor and memory subsystem to the PCI Express switch fabric comprised of one or more switch devices (embodiments are also possible without switches, however). In PCI Express, a point-to-point architecture is used. Similar to a host bridge in a PCI system, the root complex generates transaction requests on behalf of the processor, which is interconnected through a local I/O interconnect. Root complex functionality may be implemented as a discrete device, or may be integrated with the processor. A root complex may contain more than one PCI Express port and multiple switch devices can be connected to ports on the root complex or cascaded.
In order to standardize a way to share PCIe devices in a way that virtualization goals are still met, the Single-Root Input/Output Virtualization (SR-IOV) standard was introduced. SR-IOV provides a mechanism by which a single root function (such as a single Ethernet port) can appear to be multiple separate physical devices. In this manner, a port leading to a PCIe device can be shared between multiple virtual machines, thus effectively sharing the PCIe devices between the virtual machines without either virtual machine needing to be aware of the existence of the other.
An SR-IOV-capable device (such as a PCIe endpoint) can be configured to appear in the PCI configuration space as multiple functions, each with its own configuration space complete with Base Address Registers (BARs). A virtual machine manager (VMM) assigns one or more virtual functions to a virtual machine by mapping the actual configuration space of the virtual functions to the configuration space presented to the virtual machine by the VMM.
As an example, FIG. 1 depicts a standard PCIe device, having three different functions 100, 102, 104, each with its own physical resources 106, 108, 110, respectively, as well as internal routing 112, configuration resources 114, and a PCIe port 116. PCIe functionality shared by all functions is managed through function 100. It should be noted that while this figure illustrates only three functions, a PCIe device can typically support up to 256 functions.
SR-IOV introduces the concepts of physical functions and virtual functions. A physical function is a PCIe function that supports the SR-IOV capability. A virtual function is a lightweight function that is associated with a physical function but that can be assigned to a particular virtual machine. In other words, each physical function may be assigned multiple virtual functions, and then each of these multiple virtual functions can be assigned to a different virtual machine, effectively sharing the physical function without any of the other virtual machines being aware of this. All of this capability is managed through the Virtual Machine Manager (VMM). This is depicted in FIG. 2. Here, a PCIe SR-IOV capable device is shown having two physical functions 200, 202, and each physical function is shown having three virtual functions 204, 206, and 208, and then 210, 212, and 214, respectively. It should be known that two physical functions and three virtual functions per physical functions are just examples depicted in this diagram. In reality, there can be any number of physical functions (up to device limits), and each physical function can have a different number of associated virtual functions.
While SR-IOV allows multiple virtual machines within a single host to share physical resources, there is no capability to allow virtual machines across multiple hosts to share physical resources. That is why it is known as “Single-Root” IOV, because it only allows a single root complex, and thus a single host, to share resources of an attached PCIe device.
With PCIe devices expanding every year, it is now more standard to have devices, such as switches, connecting multiple hosts to multiple PCIe devices. It would be advantageous to allow these multiple hosts to share PCIe endpoint functions, because it would allow for the PCIe endpoint functions to be dynamically provisioned among the hosts to meet workload requirements. One proposed solution is known as Multi-Root Input/Output Virtualization (MR-IOV). This method has been standardized, however due to resource limitations it has not proved popular, and is barely in use. Even if one were to try and implement it on a new switch, the lack of availability of MR-IOV compatible PCIe endpoints would make such a switch virtually useless.
What is needed is a solution that allows the multi-root sharing of endpoint functions using the existing SR-IOV standard that is in use by a large number of devices, thus having the advantages of MR-IOV without needing to actually implement MR-IOV.